Consolidated toolbar during command execution - #1745

1. Pipes to interactive targets (cmd2.py:3366, command_toolbar.py:43) — the pipe process now inherits the real terminal via the new pipe_target() helper, and the toolbar suspends for the life of the pipe (held in RedirectionSavedState.toolbar_suspension, released in _restore_output). Streams with no file descriptor still fall back to PIPE. Verified under a real pty: alias | python3 -c "...isatty()" printed PIPE_ISATTY True; before the fix that command produced no output at all.

2. Ctrl-C not reaching subprocesses (command_toolbar.py:161) — POSIX now uses os.killpg(os.getpgrp(), SIGINT), matching what the terminal driver does. Verified under a pty with a child that reports its own SIGINT: post-fix CHILD_GOT_SIGINT appears and the child exits immediately; pre-fix it never fired and the child ran to completion. Windows keeps interrupt_main() — broadcasting a console control event there would hit unrelated processes on the same console. Ctrl-\ stays inert, which matches the main prompt's existing behavior; documented rather than changed.

3. Failing stop() disabling the toolbar (cmd2.py:2098) — _command_toolbar = None moved into its own finally.

4. Writes racing _pause() (command_toolbar.py) — an RLock shared by the toolbar and its ToolbarStreams serializes writes against the proxy swap. The lock is deliberately released before _thread.join(), since the toolbar thread can itself be blocked writing through a stream.

5. Post-startup failures swallowed — new _app_exited() detaches the streams when the display stops on its own, so output goes to the terminal instead of a dead proxy, and reports the error. Worth noting: the review's stated trigger doesn't hold — a get_bottom_toolbar() that raises on refresh is caught by asyncio's default handler and the thread survives. The fix targets the case where the thread genuinely dies.

6. Final flush cancelled before writing — does not reproduce. Application.run_async already awaits wait_for_cpr_responses() and any in-flight run_in_terminal futures before its loop closes (application.py:770-779). I built the exact scenario — an output with responds_to_cpr=True and a CPR future confirmed pending at pause time — and the trailing text arrived with and without the proposed fix. I reverted my drain implementation, which was adding a run_in_terminal round-trip and up to a second of CPR waiting at every command boundary for nothing. I kept the test as a regression guard and a two-line guard against Application.exit() raising in a loop callback if the app has already stopped.

7. Docs/example mismatch — do_work now uses Annotated[int, Argument(nargs="?")], so work and work 5 both work. Verified in the real app.

Three tests called _read_raw_input() after the toolbar context had exited,
so no prompt-toolkit app session was active. patch_stdout() then asked the
default app session for its output, which lazily builds a real terminal
Output: harmless Vt100_Output on POSIX, but Win32Output on Windows, which
raises NoConsoleScreenBufferError without a console screen buffer of the
kind GitHub Actions does not provide.

Bind the fixture's app session to the test pipe and recording output so
every prompt-toolkit call in these tests resolves to the test terminal
rather than the ambient one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C7nkx8kKC2mpaMGfLwXe6J

…d an embedded pager

Following prompt-toolkit’s [layout composition model](https://python-prompt-toolkit.readthedocs.io/en/3.0.52/pages/full_screen_apps.html#the-layout).

- CommandToolbar (cmd2/command_toolbar.py:101) now reuses main_session.app and its actual toolbar container, eliminating the second application and duplicated display configuration.

- The new pager (cmd2/pager.py:37) shares that application. The toolbar continues refreshing during scrolling, searching, and resizing. Colors, wrapped lines, and horizontal scrolling are supported.

- Enabling the toolbar selects the embedded pager. Set self.use_builtin_pager = False to retain your external pager.

An external pager such as less takes control of the terminal; a layout alone cannot reserve space around it. That path still temporarily suspends the toolbar.

This simplifies rendering ownership, although the embedded pager adds code.

The command UI thread and output proxy remain necessary for synchronous commands. There’s also one guarded dependency on PromptSession’s internal layout structure, verified against versions 3.0.52 and 3.0.53.

Closing the terminal's input ends the display's Application.run() with
EOFError, which _resume() swallows because it is an ordinary shutdown
rather than a failure the running command should hear about. Nothing
exercised that handler, leaving it as the one uncovered line in the new
module.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C7nkx8kKC2mpaMGfLwXe6J

page() constructed a full Pager -- a TextArea, a SearchToolbar, an ANSI
lexer and its key bindings -- and then threw it away whenever the output
fit on screen, which is a common case for ppaged(). Only the line
measurements were ever used on that path.

Move the size check into a module-level output_fits() that measures the
text directly, and build the Pager only once the output has to be
scrolled. A shared _fragments() helper does the one ANSI parse for both,
so the measurement cannot drift from what the view would render.

The fast path's size check drops from ~0.15ms to ~0.02ms per call, so it isn't a massive speedup or anything, but still cleaner.

1. Chop-mode paging no longer loses your column — cmd2/pager.py:_scroll()
The destination column now comes from the window's current horizontal_scroll when chop is set, instead of target * window_width (always 0 in chop mode)

2. A toolbar that can't start no longer takes the session down - cmd2/cmd2.py
_command_toolbar_context() catches startup failure, reports once via perror, sets a new _command_toolbar_disabled flag so later commands don't retry and re-spam and runs the command without a toolbar

3. ppaged() no longer starts a toolbar outside the command loop — it now pages inside a toolbar only when one is already running, and otherwise falls through to the external pager. The documentation has been updated to contain truthful statements in this regard.

4. use_builtin_pager documented as opt-out only — initialization.md, os.md, prompt.md and the ppaged() docstring now state that it cannot turn the embedded pager on where no toolbar is running, since the two share one display.

…g visible hyperlink text, colors, and width calculations

When ppaged() receives Rich output containing hyperlinks, Rich emits OSC 8 sequences that prompt-toolkit's ANSI parser does not understand. Their contents become visible pager text: a linked click here renders as 8;id=...;https://example.comclick here8;;. This also corrupts width calculations and searchable text.

This change strips unsupported OSC sequences before passing captured output to cmd2's new internal pager.

…toolbar.py

Also:
- Moved common code used by test_command_toolbar.py and test_pager.py to conftest.py

Since the pager is effectively its own component, it feels cleaner to have the tests in their own file.

test_command_toolbar_ui_call_reports_display_failure patched
call_soon_threadsafe on the toolbar's live event loop and swallowed every
call, not just the one it meant to drop. The patch was still installed
while asyncio.run() tore the loop down, and asyncio resolves the default
executor's join future through call_soon_threadsafe, so that report was
dropped too.

On Linux and macOS prompt_toolkit registers stdin with loop.add_reader, no
default executor is ever created, and shutdown_default_executor() returns
immediately. On Windows prompt_toolkit parks its WaitForMultipleObjects
watcher in loop.run_in_executor(), so the executor exists and the loop
waits on a future nothing resolves: 300 seconds on Python 3.12 and later,
and forever before that, where shutdown_default_executor() takes no
timeout. Windows CI went from 34s to 345s on 3.14, and hung for six hours
on 3.11.

Drop only the request made on the command thread and pass everything else
through. The display still dies with the ValueError the test asserts on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ej3Nyn6vGngxYqm9FA1Jw5

…Windows

It now has three commands, in the order your colleague should try them:

1. slow — sleeps 5 seconds. Does TOOLBAR IS HERE stay visible the whole time? This is the discriminating test: if the toolbar isn't up during a command, the real bug is that CommandToolbar never starts on Windows, and the pager is just collateral damage.

2. longout — pages 500 lines. Built-in pager ends in q: quit; the external one shows -- More --.

3. diag — dumps the guard state to real stderr (so it survives whatever the pager does to the screen).

The single most useful thing to report is the slow result plus the diag block — that alone should help narrow down the underlying root cause.

…ooting

read_input() and read_secret() each build their own PromptSession and passed no
bottom_toolbar, while _read_raw_input() suspends the main display. The toolbar
therefore vanished entirely for as long as a nested prompt was up, which is a
visible inconsistency for a bar documented as persistent.

Pass the toolbar to both sessions when the application has one, gated on
main_session.bottom_toolbar being set so an app without a toolbar does not sprout
one. Also inherit refresh_interval, so a clock or status display keeps ticking
while the nested prompt waits rather than freezing at its first render.

select() is deliberately left alone: prompt-toolkit's choice() accepts a toolbar
but has no refresh_interval, so its toolbar would go stale while the selection
sits idle.

…nd a UI-call race (#1752)

* Render test output independently of the caller's environment

Rich and cmd2 consult several environment variables when deciding whether to
emit styling, and the suite inherited them. Exporting any one of them made large
numbers of unrelated tests fail depending on who ran the suite: NO_COLOR failed
15 tests, and FORCE_COLOR and TTY_COMPATIBLE 53 each. The failures look like
product regressions, which makes them expensive to diagnose.

Neutralize them for every test. Tests that exercise these variables set them
explicitly, which still works because a test's own monkeypatching runs after the
fixture. A guard test fails if any of them reaches a test again.

* Write redirected and piped output as UTF-8

Command output is rendered by Rich and routinely contains non-ASCII, but
redirection targets and pipes were opened with the locale's encoding. On any
system whose default is not UTF-8 -- a Windows console using a legacy code page,
for instance -- redirecting output raised UnicodeEncodeError, and the user was
left with an empty file and advice to set PYTHONIOENCODING.

Open both with UTF-8 explicitly. Two tests that read redirected output back were
relying on the locale encoding for decoding as well, so they now name it too.

* Fix a race that reported a timeout for a successful UI call

_call_in_ui() polls the pending future with a 0.1s timeout and, on expiry,
re-raises when the future is already done. That branch exists because
concurrent.futures.TimeoutError is TimeoutError on Python 3.11+, so a callback
raising a timeout of its own cannot be told apart by type from the poll expiring.

Re-raising the caught exception conflates the two. When the callback completes
in the window between the poll expiring and the future being inspected, the
caller is told the call timed out even though it succeeded. Ask the future for
its outcome instead: a callback that raised a timeout still propagates it, and
one that produced a value now returns it.

Found while investigating an intermittent failure of
test_command_toolbar_ui_call_propagates_failures, which reproduced once in 60
runs before this change and not once in 120 after. The added regression test
drives the interleaving deterministically rather than relying on timing.

(cherry picked from commit 84bc19e)

* Run the pipe encoding test through this interpreter, not cat

The test piped through `cat`, which cmd.exe does not provide. On a Windows
system without Unix utilities installed it would fail before reaching the
encoding behavior it exists to check -- and Windows is exactly what the UTF-8
redirection fix targets.

Use a sys.executable pass-through instead, matching the pipe tests already in
tests/test_command_toolbar.py. Reverting either the pipe or the redirect
encoding still fails these tests.

Measured which Windows ANSI code pages can represent the box-drawing characters
Rich emits: every cp125x code page fails, covering US and Western European,
Central European, Cyrillic, Greek, Turkish, Hebrew, Arabic, Baltic and
Vietnamese systems. Only the CJK double-byte code pages survive.

Calling it a legacy code page was wrong: cp1252 is the default on a current,
fully updated Windows 11, and Python only defaults to UTF-8 mode in 3.15.

# Conflicts:
#	CHANGELOG.md

# Conflicts:
#	CHANGELOG.md
#	cmd2/cmd2.py
#	cmd2/utils.py
#	tests/conftest.py

# Conflicts:
#	CHANGELOG.md
#	docs/features/prompt.md

While a command runs, the bottom toolbar's display renders to the main
session's output from its own thread. set_window_title() only found that
display's loop through active_session, which still names the original
session when main_session is replaced after construction, so a title set
during a command could be written and flushed from the calling thread,
unsynchronized with the toolbar's rendering.

Treat the toolbar's display like a prompt: queue the title on its event
loop while it runs, and write any title whose callback the loop didn't run
once the display has stopped.

run_editor() now calls _run_shell_command() directly instead of going through do_shell(), so the
suspension on do_shell() no longer covered the editor. Suspend around _run_shell_command(), which
both callers share.

Ctrl-Z in the built-in pager, or at the command toolbar, sends SIGTSTP from the
display thread. A process-directed signal lands on the main thread, and on
Linux the display thread carries on for a few milliseconds: it restores raw
mode, reattaches its input and redraws before the job has stopped. When the
shell resumes the job it puts back the terminal modes it saved beforehand, and
nothing is left to undo that. The terminal stays cooked, keys are held until
Enter, and with a refreshing toolbar every cursor-position query is echoed as
^[[row;colR.

After sending the signal, wait for the stop to take the sending thread as
well. A stop shows as time passing while asleep; a signal that was ignored, or
discarded for an orphaned process group, shows as nothing and the wait ends on
its own.

This is the fix made in 66513f6 on stage4-handoffs-integration, without its
reserved toolbar parts. The regression test drives the pager under an
interactive bash on a pty, suspends and resumes it, and requires that it still
answers q. It fails on Linux without this change.

An error raised while the toolbar redraws during a command, most likely from
get_bottom_toolbar(), went to asyncio's default handler, which logs a traceback
to stderr. That write passes through the stdout proxy, whose redraw calls the
failing callback again, so one faulty callback printed hundreds of tracebacks
per second for as long as the command ran.

Handle errors on the display's event loop by stopping the display, which
reports the error once through the path that already existed for a display
that stops on its own. A display which has failed is not restarted for the
rest of the command, and a display which cannot come back after a pause is
reported instead of raised: that exception would otherwise fail the command
which paused it, or escape cmdloop() from command finalization.

Pausing the toolbar waits for the refresh in progress, so a command holding a
lock which get_bottom_toolbar() acquires must not prompt, page or run a
subprocess while it holds it. Say so where the documentation recommends a lock.

Also note at the key_bindings swap that prompt-toolkit caches merged bindings
per focused window and does not see the attribute being reassigned.

prompt-toolkit draws the toolbar starting at the cursor, and erases it from the
first column of that row before printing anything more above it. Output which
stopped in the middle of a line was erased along with the toolbar: a line
written as 'Working...' and finished later lost its first part, a progress line
redrawn with carriage returns vanished once it was complete, and a last line
without a newline was lost whenever the display redrew before it stopped.

End an unfinished line on the screen so the toolbar always starts on a row of
its own, and when more output arrives move back up and write the unfinished
line again before it, which puts the cursor and any style it set where they
were. A line unfinished when the display stops stays ended, so the next prompt
starts on a new line, as it does with reserved rendering.

Stopping the display after an error draws a last frame and fires after_render
again. A handler which fails there replaced the error already recorded, so the
report named a consequence instead of the cause.

That is how a race in the pager navigation test surfaced on CI. Its
after_render check read the terminal's size after another thread had resized
it, so a frame drawn at the old size was judged against the new one. asyncio
used to log and ignore that assertion. Now it stops the display, and the check
failed again on the last frame with an unrelated AttributeError. Judge each
frame against the size it was drawn at.